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Program Description^ 


Bright Beginnings 

Bright Beginnings is an early childhood curriculum, based In part 
on High/Scope® and Creative Curricuium®, with an additional 
emphasis on literacy skills. The curriculum consists of nine 
thematic units designed to enhance children’s cognitive, social, 
emotional, and physical development, and each unit includes 


concept maps, literacy lessons, center activities, and home 
activities. Special emphasis is placed on the development of 
early language and literacy skills, and parent involvement is 
a key component of the program. 


Research^ 


No studies of Bright Beginnings meet What Works Clearinghouse 


print knowledge, phonological processing, and math. No studies 



(WWC) evidence standards, but one study meets WWC evidence that meet WWC evidence standards with or without reservations 

standards with reservations. The one study included 198 students examined the effectiveness of Bright Beginnings in the early 

from 14 public preschool classrooms in Tennessee.® reading and writing or the cognition domains. 



Based on this one study, the WWC considers the extent of 
evidence for Bright Beginnings to be small for oral language, 

Eff6CtiV6n6SS Bright Beginnings was found to have potentially positive effects on print knowledge and no discernible effects on oral language. 



phonological processing, and math. 














Oral 

language 


Print 

knowledge 


Phonological 

processing 


Early reading 
and writing 


Cognition 


Math 


Rating of effectiveness 


No discernible 
effects 


Potentially 
positive effects 


No discernible 
effects 


na 


na 


No discernible 
effects 


Improvement index'^ 


Average: +4 
percentile points 


Average: -i-12 
percentile points 


Average: -3 
percentile points 


na 


na 


Average: +4 
percentile points 




Range: +4 to +5 
percentile points 


Range: +7 to +15 
percentile points 


na 


na 


na 


Range: -1 to +6 
percentile points 
na = not applicable 



1. The descriptive information for this program was obtained from pubiiciy avaiiabie sources: the program’s website (http://www.cms.k12.nc.us/cmsde- 
partments/ci/pre-kservices/Pages/defauit.aspx, downioaded August 2008) and the iiterature reviewed for this report. The WWC requests deveiopers 
to review the program description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive information for 
this program is beyond the scope of this review. 

2. The studies in this report were reviewed using WWC Evidence Standards, Version 1.0 (see the WWC Standards). 

3. The evidence presented in this report is based on avaiiabie research. Findings and conclusions may change as new research becomes available. 

4. These numbers show the average and range of student-level improvement indices for all findings across the study (studies). 
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Absence of conflict 
of interest 



The PCER Consortium (2008) study summarized in this inter- 
vention report had numerous contributors, including staff of 
Mathematica Policy Research, Inc. (MPR). Because the principal 



investigator for the WWC review of Early Childhood Education is 
also a MPR staff member, the study was rated by Chesapeake 
Research Associates, who also prepared the intervention report. 



Additionai program 
information 



Developer and contact 

Developed by former superintendent of Charlotte-Mecklenburg 
Schools, Eric Smith, in conjunction with district staff and local 
businesses. Bright Beginnings Office: Family Application Center, 
700 Marsh Road, Charlotte, NC 28209. Email: prek@cms.k12.nc.us. 
Web: http://www.cms.k12.nc.us/cmsdepartments/ci/pre-kservices/ 
Pages/default.aspx. Telephone: (980) 343-3797. 

Scope of use 

Bright Beginnings was funded by the Charlotte-Mecklenburg 
district’s Title I grant, and has been used primarily in the 
Charlotte-Mecklenburg school system, in five prekindergarten 
centers and 14 elementary schools. Recently, the program has 
been implemented in other districts as well. 

Teaching 

The Bright Beginnings program is based on the High/Scope® 
and Creative Curriculum® models, with an additional emphasis 
on the development of early literacy skills. The curriculum is 
designed to create a child-centered, literacy-focused program 
that is relevant to the developmental needs of young children 



and addresses their cognitive, social, emotional, and physical 
development. Bright Beginnings consists of nine units: language 
and literacy, mathematics, social and personal development, 
healthful living, scientific thinking, social studies, creative arts, 
physical development, and technology. Active exploration and 
interaction with other students, adults, and materials are impor- 
tant components of the Bright Beginnings model. As children 
participate in a variety of Bright Beginnings activities, they are 
continually monitored by teachers to assess their progress. 
Another feature of the Bright Beginnings program is an effort 
to engage parents; they are required to sign a parent-school 
partnership agreement affirming their active participation in their 
children’s education. 

Cost 

District cost estimates, based on use in the Charlotte- 
Mecklenburg school system, range from $9.8 million for 1,672 
Bright Beginnings students in the first cohort (1997-98) to $16.7 
million for 3,020 Bright Beginnings students in the sixth cohort 
(2002-03), resulting in a cost of approximately $5,500 to $5,800 
per student. 



Research 



Five studies reviewed by the WWC investigated the effects of 
Bright Beginnings. One study (PCER Consortium, 2008) is a ran- 
domized controlled trial design that had non-random allocations 
after random assignment, but the analytic groups were shown 
to be equivalent, so the study meets WWC evidence standards 
with reservations. No studies are randomized controlled trials or 
quasi-experimental designs that meet WWC evidence standards. 
The remaining four studies do not meet either WWC evidence 
standards or eligibility screens. 



Meets evidence standards with reservations 

One study reviewed by the WWC (PCER Consortium, 2008) 
assessed the effectiveness of Bright Beginnings as part of the 
Preschool Curriculum Evaluation Research effort.® The PCER 
Consortium (2008) randomly assigned 36 classrooms to three 
conditions (Bright Beginnings, Creative Curriculum®, and con- 
trol), randomly selecting 21 classrooms for the national evalu- 
ation. After the pilot year, 8 classrooms dropped out (2 Bright 
Beginnings, 3 Creative Curriculum®, and 3 control) and were 



5. The Preschool Curriculum Evaluation Research Consortium (2008) evaluated a total of 14 preschool curricula, including Bright Beginnings, in comparison 
to respective control conditions. 
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Research (continued) replaced with classrooms from the original 36. For this study, 

14 classrooms were used (including the original and replacement 
classrooms), and baseline equivalence on pretests was estab- 
lished for the Bright Beginnings and control children. Data were 
collected on 198 children (98 Bright Beginnings and 100 control). 
Just over half of the children (51%) were male, 82% were Cau- 
casian, and 23% were reported to have a disability. Pretest data 
were collected in the fall and posttest data were collected in the 
spring of the preschool year. The study investigated effects on 
oral language, print knowledge, phonological processing, and 
math. The control condition varied across sites and included 
teacher-developed, nonspecific curricula with a focus on basic 
school readiness. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 
Standards Handbook, Appendix G). The extent of evidence 
takes into account the number of studies and the total sample 
size across the studies that meet WWC evidence standards with 
or without reservations.® 

The WWC considers the extent of evidence for Bright Begin- 
nings to be small for oral language, print knowledge, phonological 
processing, and math. No studies that meet WWC evidence 
standards with or without reservations examined the effective- 
ness of Bright Beginnings in the early reading and writing or the 
cognition domains. 



Effectiveness Findings 

The WWC review of interventions for early childhood education 
addresses child outcomes in six domains: oral language, print 
knowledge, phonological processing, early reading and writing, 
cognition, and math. The study included in this report covers four 
domains: oral language, print knowledge, phonological process- 
ing, and math. The findings below present the authors’ estimates 
and WWC-calculated estimates of the size and the statistical 
significance of the effects of Bright Beginnings on children.^ 

Orai Language. The PCER Consortium (2008) analyzed the 
effectiveness of Bright Beginnings on oral language using the 
Peabody Picture Vocabulary Test (PPVT-III) and the Test of Lan- 
guage Development (TOLD). The authors report, and the WWC 



confirms, that differences between the Bright Beginnings and 
control groups are not statistically significant or substantively 
important on any of these measures. According to WWC criteria, 
this study shows no discernible effects on oral language. 

Print Knowledge. The PCER Consortium (2008) analyzed the 
effectiveness of Bright Beginnings on the Test of Early Reading 
Ability (TERA-3), the Woodcock-Johnson III (WJ-III) Letter-Word 
Identification subtest, and the WJ-III Spelling subtest.® The study 
authors report a statistically significant positive effect of Bright 
Beginnings on the TERA-3, but according to WWC calculations, 
after correcting for multiple comparisons, the effect was not 
statistically significant. The study authors report, and the WWC 
confirms, there was no statistically significant effect on the WJ-III 



6. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the Intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for Bright Beginnings is in Appendix A6. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate 
the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, 
Appendix D for multiple comparisons. No correction for clustering was needed for the study by the PCER Consortium (2008) because their analysis 
corrected for clustering by using hierarchical linear modeling (HLM). A correction for multiple comparisons was needed for the PCER Consortium (2008) 
study, so the significance levels in this report may differ from those reported in the original study. 

8. By name, the TERA-3 sounds like it should be captured under the early reading and writing domain; however, the description of the measure identi- 
fies constructs that are pertinent to print knowledge, such as knowing the alphabet, understanding print conventions, and environmental print. More 
detailed explanations of the measures in each domain can be found in Appendices 2.1-2.4. 
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Effectiveness (continued) 



Letter-Word Identification subtest or on the WJ-III Speiiing subtest. 
Although the effects are not statistically significant, the PCER 
Consortium found, and the WWC confirmed, that the effects on 
theTERA-3 and on the WJ-III Letter-Word Identification subtest are 
large enough to be considered substantively important according 
to WWC criteria (that is, at least 0.25). According to WWC criteria, 
the study shows potentially positive effects on print knowledge. 

Phonological Processing. The PCER Consortium (2008) 
analyzed the effectiveness of Bright Beginnings on phonological 
processing using the Preschool Comprehensive Test of Phono- 
logical and Print Processing. The authors report, and the WWC 
confirms, that differences between the Bright Beginnings and 
control groups are not statistically significant or substantively 
important on this measure. According to WWC criteria, this study 
shows no discernible effects on phonological processing. 

Math. The PCER Consortium (2008) analyzed the effective- 
ness of Bright Beginnings on math using the WJ-III Applied 



Problems subtest, the Composite Score from the Child Math 
Assessment-Abbreviated, and Shape Composition. The authors 
report, and the WWC confirms, that differences between the 
Bright Beginnings and control groups are not statistically 
significant or substantively important on any of these measures. 
According to WWC criteria, this study shows no discernible 
effects on math. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effectiveness 
takes into account four factors: the quality of the research design, 
the statistical significance of the findings, the size of the difference 
between participants in the intervention and the comparison condi- 
tions, and the consistency in findings across studies (see the WWC 
Procedures and Standards Handbook, Appendix E). 



The WWC found 
Bright Beginnings to have 
potentially positive effects 
on print knowledge and no 
discernible effects on oral 
language, phonological 
processing, and math 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see WWC Procedures 
and Standards Handbook, Appendix F). The improvement index 
represents the difference between the percentile rank of the aver- 
age student in the intervention condition and the percentile rank of 
the average student in the comparison condition. Unlike the rating 
of effectiveness, the improvement index is entirely based on the 
size of the effect, regardless of the statistical significance of the 
effect, the study design, or the analysis. The improvement index 
can take on values between -50 and +50, with positive numbers 
denoting favorable results for the intervention group. 

Based on one study (PCER Consortium, 2008), the average 
improvement index of Bright Beginnings on two measures of 
oral language is +4 percentile points with a range from +4 to +5 



percentile points across findings. For print knowledge, the 
average improvement index on three measures is +12 percentile 
points with a range from +7 to +15 percentile points across 
findings. For phonological processing, the improvement index 
on one outcome measure is -3 percentile points. For math, the 
average improvement index on three measures is +4 percentile 
points with a range from -1 to +6 percentile points. 

Summary 

The WWC reviewed five studies of Bright Beginnings. One of 
these studies meets WWC evidence standards with reservations; 
the remaining four studies do not meet either WWC evidence 
standards or eligibility screens. Based on the one study, the 
WWC found potentially positive effects on print knowledge and 
no discernible effects on oral language, phonological process- 
ing, and math. The conclusions presented in this report may 
change as new research emerges. 
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Appendix 



Appendix A1 Study characteristics: Preschooi Curricuium Evaiuation Research Consortium, 2008 (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Preschool Curriculum Evaluation Research (PCER) Consortium (2008). Bright Beginnings and Creative Curricuium-. Vanderbiit University. In Effects of Preschool Curricuium Programs 
on School Readiness (pp. 41-54). Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education. 


Participants 


The study, conducted during the 2003/04 and 2004/05 school years, included three intervention groups: Bright Beginnings, Creative CurriculurrP, and a control group. Thirty- 
six full-day prekindergarten classrooms in 28 public preschool programs in Tennessee were recruited and blocked into groups of three by matching them on composite factors 
for demographic characteristics (urban/rural, percent of races other than White) and achievement (percent free lunch, reading, language, mathematics, and science achieve- 
ment scores). Within each block, one preschool was randomly assigned to Bright Beginnings, one to Creative Curriculurrf , and one to the control group. The manuscript notes 
that the researchers randomly assigned the classrooms to three conditions; however, all classrooms within a preschool were assigned to the same condition. Subsequent to 
randomization, 21 of the 36 classrooms (7 from each of the three groups) were randomly selected to participate in the national PCER study of Bright Beginnings and Creative 
CurriculurrP . All 36 classrooms participated in the local investigator’s pilot study during the first year. Eollowing the pilot year, and prior to starting the national PCER study, 8 
of the 21 originally assigned classrooms dropped out of the study, leaving 5 Bright Beginnings, 4 Creative CurricuiurrP, and 4 control classrooms. The 8 dropout classrooms 
were replaced by randomly selecting 8 from the 15 classrooms that had not been selected to participate in the national PCER study, including 2 Bright Beginnings, 3 Creative 
CurriculurrP , and 3 control classrooms, restoring the sample of classrooms to 7 in each of the three intervention groups. The evaluation of Bright Beginnings included 14 
classrooms (7 Bright Beginnings and 7 control) and a total of 208 children at baseline (1 03 Bright Beginnings and 1 05 control), while the analysis sample included 98 Bright 
Beginnings cUWdm and 100 control children. Pretest differences between the treatment and comparison groups were not statistically significant. At baseline, children in 
the study averaged 4.5 years of age; 51% were male; and 11% were Elispanic, and 82% were White. A higher percentage of parents in the control group reported that their 
child had an Individualized Education plan relative to those assigned to Bright Beginnings (33 percent vs. 13 percent), a difference that was statistically significant, but did not 
exceed the 25% upper limit on acceptable baseline differences between groups that is indicated in the WWC Early Childhood Education protocol. 


Setting 


The Bright Beginnings study ms conducted in prekindergarten classrooms in 14 public schools (7 Bright Beginnings and 7 control) from 7 county school districts in Tennessee. 


Intervention 


Bright Beginnings is an integrated curriculum with a focus on language and early literacy, based in part on the High/Scop^ and Creative CurriculurrP models, with an added 
focus on skills designed to promote school literacy. Bright Beginnings includes nine curriculum units: language and literacy, mathematics, social and personal development, 
healthful living, scientific thinking, social studies, creative arts, physical development, and technology. In the PCER study, each classroom’s fidelity to the curriculum was rated 
on a four-point scale, ranging from “not at all” (0) to “high” (3). The average score for the Bright Beginnings classrooms was 1 .88 on the measure. 


Comparison 


Control teachers used teacher-developed, nonspecific curricula with a focus on basic school readiness. Their classrooms were rated with the same fidelity measure used in 
the Bright Beginnings classrooms, which ranged from 0 to 3. The average score for the control classrooms was 2.0. 


Primary outcomes 
and measurement 


The outcome domains assessed were children’s oral language, print knowledge, phonological processing, and math. Oral language was assessed with the Peabody Picture 
Vocabulary Test-Ill (PPVT-III) and the Test of Language Development-Primary III (T0LD-P:3) Grammatic Understanding subtest. Print knowledge was assessed with the Test of 
Early Reading Ability-Ill (TERA-3), the Woodcock-Johnson III (WJ-III) Letter-Word Identification subtest, and the WJ-III Spelling subtest. Phonological processing was assessed 
with the Preschool Comprehensive Test of Phonological and Print Processing (Pre-CTOPPP) Elision subtest. Math was assessed with the WJ-III Applied Problems subtest, the 
Child Math Assessment-Abbreviated (CMA-A), and the Shape Composition task. Eor a more detailed description of these outcome measures, see Appendices A2.1-2.4. 


Staff/teacher training 


Bright Beginnings teachers received 2.5 days of curriculum training prior to the start of the prekindergarten year. Onsite consultation to teachers was provided four times dur- 
ing the school year: twice by trained Tennessee staff members and twice by curriculum trainers. Consultation visits typically included a classroom observation, an opportunity 
for teachers to ask questions about the curriculum, and implementation feedback from the trainer. 
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Appendix A2.1 Outcome measures for the oral language domain 



Outcome measure 


Description 


Peabody Picture Vocabulary 
Test-3rd Edition (PPVT-III) 


A standardized measure of children's receptive vocabulary in which children demonstrate understanding of a spoken word by pointing to a picture that best represents the 
meaning (as cited in PCER Consortium, 2008). 


Test of Language 
Development-Primary III 
(T0LD-P;3) Grammatic 
Understanding subtest 


A standardized measure of children's ability to comprehend the meaning of sentences by selecting pictures that most accurately represent the sentence (as cited in PCER 
Consortium, 2008). 



Appendix A2.2 Outcome measures for the print knowledge domain 



Outcome measure 


Description 


Test of Early Reading 
Ability III (TERA-3) 


A standardized measure of children's developing reading skills with three subtests: alphabet, conventions, and meaning (as cited in PCER Consortium, 2008).'' 


Woodcock- Johnson III 
(WJ-III) Letter-Word 
Identification subtest 


A standardized measure of identification of letters and reading of words (as cited in PCER Consortium, 2008). 


Woodcock-Johnson III 
(WJ-III) Spelling subtest 


A standardized measure that assesses children's prewriting skills, such as drawing lines, tracing, and writing letters (as cited in PCER Consortium, 2008). 



1. By name, this measure sounds like it should be captured under the early reading and writing domain; however, the description of the measure identifies constructs that are pertinent to print 
knowledge, such as knowing the alphabet, understanding print conventions, and environmental print. 

Appendix A2.3 Outcome measures for the phonological processing domain 



Outcome measure 


Description 


Preschool Comprehensive 
Test of Phonological and 
Print Processing (Pre- 
CTOPPP), Elision subtest 


A measure of children's ability to identify and manipulate sounds in spoken words, using word prompts and picture plates for the first nine items and word prompts only for 
later items (as cited in PCER Consortium, 2008). 
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Appendix A2.4 Outcome measures for the math domain 



Outcome measure 


Description 


Woodcock-Johnson III 
(WJ-III) Applied 
Problems subtest 


A standardized measure of children's ability to solve numerical and spatial problems, presented verbally with accompanying pictures of objecfs (as cifed in PCER 
Consortium, 2008). 


Child Math Assessment- 
Abbreviated (CMA-A) 
Composite Score 


The average of four subscales: (1) solving addifion and subtraction problems using visible objects, (2) constructing a set of objects equal in number to a given set, 
(3) recognizing shapes, and (4) copying a pattern using objects that vary in color and identity from the model pattern (as cited in PCER Consortium, 2008). 


Building Blocks, Shape 
Composition task 


Modified for PCER from the Building Blocks assessment tools. Children use blocks to fill in a puzzle and are assessed on whether they fill the puzzle without gaps 
or hangovers (as cited in PCER Consortium, 2008). 
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Appendix A3.1 Summary of study findings inciuded in the rating for the orai ianguage domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

groups group 


Mean 

difference'* 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008 (meets standards with reservations)^ 



PPVT-III Preschoolers 


14/195 


96.31 

(14.71) 


93.93 

(15.37) 


2.38 


0,13 


ns 


-h5 


T0LD-P:3 Grammatic Preschoolers 

Understanding subtest 


14/197 


9.60 

(2.95) 


9.11 

(2.73) 


0.49 


0.09 


ns 


-h4 


Domain average for oral language® 










0.11 


na 


+4 



ns = not statistically significant 

na = not applicable 

PPVT-III = Peabody Picture Vocabulary Test-Ill 

T0LD-P:3 = Test of Language Development Primary, Third Edition 

1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the oral language domain. Follow-up findings from PCER Consortium (2008) are 
not included in these ratings but are reported in Appendix A4.1. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 

The improvement index can take on values between -50 and -i-50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 

sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections 

were needed because the analysis corrected for clustering by using hierarchical linear modeling (HLM), and no impacts were statistically significant. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the print knowiedge domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

groups group 


Mean 

difference'* 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008 (meets standards with reservations)^ 



TERA-3 


Preschoolers 


14/198 


91.41 

(15.91) 


87.98 

(14.71) 


3.43 


0,39 


ns 


+15 


WJ-III Letter-Word Identification 
subtest 


Preschoolers 


14/198 


106.06 

(14.97) 


97.21 

(13.03) 


8.85 


0,35 


ns 


+14 


WJ-III Spelling subtest 


Preschoolers 


14/198 


95.75 

(12.46) 


90.94 

(12,98) 


4.81 


0.18 


ns 


+7 


Oomain average for print knowledge® 










0.31 


na 


+12 



ns = not statistically significant 

na = not applicable 

TERA-3 = Test of Early Reading Ability III 

WJ-III = Woodcock-Jobnson III 

1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the print knowledge domain. Follow-up findings from PCER Consortium (2008) 
are not included in these ratings but are reported in Appendix A4.2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Flandbook, Appendix C for clustering and WWC Procedures and Standards Flandbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no correction 
for clustering was needed because the analysis corrected for clustering by using hierarchical linear modeling (FILM), but a correction for multiple comparisons was necessary. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A3.3 Summary of study findings inciuded in the rating for the phonoiogicai processing domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

groups group 


Mean 

difference'* 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008 (meets standards with reservations)^ 



Pre-CTOPPP Elision subtest Preschoolers 


14/198 


10.02 

(4.50) 


10,38 

(4,78) 


-0,36 


-0,07 


ns 


-3 


Oomain average for phonological processing® 










-0.07 


na 


-3 



ns = not statistically significant 
na = not applicable 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the phonological processing domain. Follow-up findings from PCER Consortium 
(2008) are not included in these ratings but are reported in Appendix A4.3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 

The improvement index can take on values between -50 and -i-50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 

sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Flandbook, Appendix C for clustering and WWC Procedures and Standards Flandbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections 

were needed because the analysis corrected for clustering by using hierarchical linear modeling (FILM), and no impacts were statistically significant. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A3.4 Summary of study findings inciuded in the rating for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

groups group 


Mean 

difference'* 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008 (meets standards with reservations)^ 



WJ-III Applied Problems subtest 


Preschoolers 


14/198 


100.69 

(14,68) 


96.48 

(16.69) 


4.21 


0.16 


ns 


-f6 


CMA-A Composite 


Preschoolers 


14/198 


0,57 

(0.25) 


0.53 

(0.27) 


0.04 


0.14 


ns 


-f6 


Shape Composition 


Preschoolers 


14/198 


1.82 

(0.93) 


1.85 

(0.91) 


-0.03 


-0,03 


ns 


-1 


Oomain average for math® 












0.09 


na 


+4 



ns = not statistically significant 

na = not applicable 

WJ-III = Woodcock-Johnson III 

CMA-A = Child Math Assessment - Abbreviated 

1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math domain. Follow-up findings from PCER Consortium (2008) are not 
included in these ratings but are reported in Appendix A4.4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 

The improvement index can take on values between -50 and -r50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 

sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Flandbook, Appendix C for clustering and WWC Procedures and Standards Flandbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections 

were needed because the analysis corrected for clustering by using hierarchical linear modeling (FILM), and no impacts were statistically significant. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A4.1 Summary of follow-up findings for the oral language domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size^ 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

group'* group 


Mean 

difference^ 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index® 



PCER Consortium, 2008 (meets standards with reservations)^ 



PPVT-III 


Kindergarteners 


nr/203 


98.43 

(10.83) 


97.21 

(13.74) 


1,22 


0.07 


ns 


-f3 


T0LD-P:3 Grammatic 
Understanding subtest 


Kindergarteners 


nr/203 


10.73 

(2.91) 


9.91 

(2.93) 


0.82 


0.16 


ns 


-f6 



ns = not statistically significant 

nr = not reported 

PPVT-III = Peabody Picture Vocabulary Test-Ill 

T0LD-P:3 = Test of Language Development Primary, Third Edition 

1. This appendix presents follow-up findings considered for measures that fall in the oral language domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3.1. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The PCER Consortium (2008) study included 134 kindergarten classrooms across all three conditions in this study (Bright Beginnings, Creative Curriculum®, and control). The number of class- 
rooms for Bright Beginnings and the control group is likely about two-thirds of the total. 

4. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -i-50, with positive numbers denoting favorable results for the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas 
the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D 
for multiple comparisons. In the case of PCER Consortium (2008), no corrections were needed because the analysis corrected for clustering by using hierarchical linear modeling (HLM), and no 
impacts were statistically significant. 
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Appendix A4.2 Summary of follow-up findings for the print knowledge domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size^ 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

group'* group 


Mean 

difference^ 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index® 



PCER Consortium, 2008 (meets standards with reservations)^ 



TERA-3 


Kindergarteners 


nr/203 


93.35 

(16.02) 


93.99 

(17.75) 


-0.64 


-0.07 


ns 


-3 


WJ-III Letter-Word Identification 
subtest 


Kindergarteners 


nr/204 


106.12 

(10.67) 


103.96 

(13.41) 


2.16 


0.09 


ns 


-h4 


WJ-III Spelling subtest 


Kindergarteners 


nr/204 


102.12 


100.57 


1.55 


0.06 


ns 


+2 



(12.09) (15.15) 

ns = not statistically significant 

nr = not reported 

TERA-3 = Test of Early Reading Ability III 

WJ-III = Woodcock-Johnson III 

1. This appendix presents follow-up findings for measures that fall in the print knowledge domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The PCER Consortium (2008) study included 134 kindergarten classrooms across all three conditions in this study (Bright Beginnings, Creative Curriculum®, and control). The number of class- 
rooms for Bright Beginnings and the control group is likely about two-thirds of the total. 

4. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -i-50, with positive numbers denoting favorable results for the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas 
the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D 
for multiple comparisons. In the case of PCER Consortium (2008), no corrections were needed because the analysis corrected for clustering by using hierarchical linear modeling (HLM), and no 
impacts were statistically significant. 
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Appendix A4.3 Summary of follow-up findings for the phonological processing domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size^ 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

groups group 


Mean 

difference^ 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index® 







PCER Consortium, 2008 (meets standards with reservations)® 








CTOPP Elision subtest 


Kindergarteners 


nr/203 4.34 4.30 0.04 

(2.76) (3.27) 


0.01 


ns 


0 



ns = not statistically significant 
nr = not reported 

CTOPP = Comprehensive Test of Phonological Processing 



1. This appendix presents follow-up findings for measures that fall in the phonological processing domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3.3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The PCER Consortium (2008) study included 134 kindergarten classrooms across ail three conditions in this study (Bright Beginnings, Creative Curriculum'^, and control). The number of class- 
rooms for Bright Beginnings and the control group is likely about two-thirds of the total. 

4. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas 
the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D 
for multiple comparisons. In the case of PCER Consortium (2008), no corrections were needed because the analysis corrected for clustering by using hierarchical linear modeling (HLM), and no 
impacts were statistically significant. 
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Appendix A4.4 Summary of follow-up findings for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size^ 
(classrooms/ 
children) 


Bright 

Beginnings Comparison 

group'* group 


Mean 

difference^ 

(Bright 

Beginnings- 

comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index® 



PCER Consortium, 2008 (meets standards with reservations)^ 



WJ-III Applied Problems subtest 


Kindergarteners 


nr/204 


103.21 

(12.77) 


99.88 

(16.18) 


3.33 


0.13 


ns 


-h5 


CMA-A Composite 


Kindergarteners 


nr/203 


0.71 

(0.17) 


0.69 

(0.18) 


0.02 


0.07 


ns 


-h3 


Shape Composition 


Kindergarteners 


nr/204 


2.49 


2.36 


0.13 


0.15 


ns 


-h6 



(072) (089) 

ns = not statistically significant 

nr = not reported 

WJ III = Woodcock-Johnson III 

CMA-A = Child Math Assessment - Abbreviated 

1. This appendix presents follow-up findings for measures that fall in the math domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3.4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The PCER Consortium (2008) study included 134 kindergarten classrooms across all three conditions in this study (Bright Beginnings, Creative Curriculum®, and control). The number of class- 
rooms for Bright Beginnings and the control group is likely about two-thirds of the total. 

4. In PCER Consortium (2008), the treatment group mean equals the sum of the unadjusted control group mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. In the case of PCER Consortium (2008), the mean differences 
are covariate-adjusted. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors (Cohen’s d based on a repeated measures analysis). 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -i-50, with positive numbers denoting favorable results for the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas 
the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D 
for multiple comparisons. In the case of PCER Consortium (2008), no corrections were needed because the analysis corrected for clustering by using hierarchical linear modeling (HLM), and no 
impacts were statistically significant. 
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Appendix A5.1 Bright Beginnings rating for the orai ianguage domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of orai ianguage, the WWC rated Bright Beginnings as having no discernibie effects. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either positive or negative. 

Met. One study of Bright Beginnings showed no statisticaiiy significant or substantiveiy important effects, either positive or negative, on orai ianguage. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Only one study of Bright Beginnings was included in this review and it showed no statistically significant or substantiveiy important 
positive effects on orai language. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on oral language. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on oral language. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on oral language. No studies 
showed indeterminate effects and no studies showed statistically significant or substantively important positive effects. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive or negative effects on oral language. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive or negative effects on oral language. 



(continued) 
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Appendix A5.1 Bright Beginnings rating for the orai ianguage domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important negative effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on oral language. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important pos/f/Ve effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on oral language. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1. Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. One study of Bright Beginnings showed no statistically significant negative effects on oral language. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on oral language. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5.2 Bright Beginnings rating for the print knowiedge domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of print knowledge, the WWC rated Bright Beginnings as having potentiaiiy positive effects. The remaining ratings (mixed effects, no dis- 
cernibie effects, potentiaiiy negative effects, negative effects) were not considered, as Bright Beginnings was assigned the highest appiicabie rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study of Bright Beginnings showed a substantively important positive effect on print knowledge. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing 
indeterminate effects than showing statistically significant or substantively important pos/f/Ve effects. 

Met. No study of Bright Beginnings showed a statistically significant or substantively important negative effect on print knowledge. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Only one study of Bright Beginnings was included in this review. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies of Bright Beginnings showed statistically significant or substantively important negative effects on print knowledge. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5.3 Bright Beginnings rating for the phonoiogicai processing domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of phonoiogicai processing, the WWC rated Bright Beginnings as having no discernibie effects. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either positive or negative. 

Met. One study of Bright Beginnings showed no statisticaiiy significant or substantively important effects, either positive or negative, on 
phonological processing. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. There was oniy one study of Bright Beginnings and it showed no statisticaiiy significant or substantively important positive effects 
on phonoiogicai processing. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on phonological processing. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on phonological processing. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing 
indeterminate effects than showing statistically significant or substantively important pos/f/Ve effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on phonological processing. 
No studies showed indeterminate effects and no studies showed statistically significant or substantively important positive effects. 



(continued) 
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Appendix A5.3 Bright Beginnings rating for the phonoiogicai processing domain (continued) 



Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive or negative effects on 
phonological processing. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than 
showing a statistically significant or substantively important effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive or negative effects on 
phonological processing. 

Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important negative effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on 
phonological processing. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important pos/f;Ve effect, or more studies showing statistically significant or 
substantively important negative effects than showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on phonological processing. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1. Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. One study of Bright Beginnings showed no statistically significant negative effects on phonological processing. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on phonological processing. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5.4 Bright Beginnings rating for the math domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of math, the WWC rated Bright Beginnings as having no discernibie effects. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either positive or negative. 

Met. One study of Bright Beginnings showed no statisticaiiy significant or substantively important effects, either positive or negative, on math. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. There was only one study of Bright Beginnings and it showed no statistically significant or substantively important positive effects on math. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on math. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on math. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing 
indeterminate effects than showing statistically significant or substantively important pos/f/Ve effects. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on math. No studies showed 
indeterminate effects and no studies showed statistically significant or substantively important positive effects. 



(continued) 
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Appendix A5.4 Bright Beginnings rating for the math domain (continued) 



Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on math. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on math. 

Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important negative effect. 

Not met. One study of Bright Beginnings showed no statistically significant or substantively important negative effects on math. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important pos/f/Ve effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on math. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1. Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. One study of Bright Beginnings showed no statistically significant negative effects on math. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. One study of Bright Beginnings showed no statistically significant or substantively important positive effects on math. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Sample size 
Preschool classrooms 


Students 


Extent of evidence^ 


Oral language 


1 


14 




197 


Small 


Print knowledge 


1 


14 




198 


Small 


Phonological processing 


1 


14 




198 


Small 


Early reading and writing 


0 


na 




na 


na 


Cognition 


0 


na 




na 


na 


Math 


1 


14 




198 


Small 



na = not applicable/not studied 

1. A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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